Papers with language and vision
Connecting Language and Vision to Actions (P18-5)
Copied to clipboard
| Challenge: | Recent advances in language and vision have made incredible progress in describing images and interacting with visual content in a physical or embodied environment. |
| Approach: | This tutorial will provide an overview of the growing number of multimodal tasks and datasets that combine textual and visual understanding. |
| Outcome: | This tutorial will review the state-of-the-art approaches to selected tasks such as image captioning, visual question answering and visual dialog. |
An Explainable Toolbox for Evaluating Pre-trained Vision-Language Models (2022.emnlp-demos)
Copied to clipboard
| Challenge: | Existing studies evaluate VLP models by comparing the fine-tuned downstream task performance with the average downstream task accuracy. |
| Approach: | They propose a toolbox for evaluating Vision-Language Pretraining (VLP) models. |
| Outcome: | The proposed toolbox provides the preliminary datasets that deepen the image-texting ability of a VLP model. |
Visio-Linguistic Brain Encoding (2022.coling-1)
Copied to clipboard
| Challenge: | Existing studies have failed to explore co-attentive multi-modal modeling for visual and text reasoning. |
| Approach: | They propose to use image and multi-modal Transformers to reconstruct fMRI brain activity . they use two popular datasets to study visual and text reasoning . |
| Outcome: | The proposed model outperforms existing models on two popular datasets . the results raise the question whether visual processing is affected implicitly by linguistic processing . |
VLIS: Unimodal Language Models Guide Multimodal Language Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing vision-language models face challenges in tasks that require complex linguistic understanding. |
| Approach: | They propose a framework that combines visual conditioning and linguistic understanding of unimodal text-only language models without further training to improve vision-language models. |
| Outcome: | The proposed framework improves vision-language models on diverse tasks including commonsense understanding and complex text generation. |
Word Representation Learning in Multimodal Pre-Trained Transformers: An Intrinsic Evaluation (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing models for linguistic representations of words are based on information extracted from large text corpora, and the sensory-motor experiences humans have with the world play an important role in determining word meaning. |
| Approach: | They propose to use contextualized word representations to learn semantic representations of words that align with human semantic intuitions. |
| Outcome: | The proposed models are shown to be more efficient on concrete word pairs than on abstract ones. |
An Evaluation of Image-Based Verb Prediction Models against Human Eye-Tracking Data (N18-2)
Copied to clipboard
| Challenge: | Recent research in language and vision has developed models for predicting and disambiguating verbs from images. |
| Approach: | They propose a verb prediction model and visual sense disambiguation model for verbs . they ask whether the image regions a model identifies as salient correlate with human intuitions about visual verbs. |
| Outcome: | The proposed model can predict verbs from images, but it is unclear to what extent it captures human intuitions about visual verbs. |
Visual Referring Expression Recognition: What Do Systems Actually Learn? (N18-2)
Copied to clipboard
| Challenge: | Existing systems for referring expression recognition ignore linguistic structure, instead relying on shallow correlations introduced by unintended biases in the data selection and annotation process. |
| Approach: | They propose to use a system trained on the input image without the input referring expression to achieve a precision of 71.2% in top-2 predictions. |
| Outcome: | The proposed model can achieve 71.2% accuracy on the input image without the input referring expression and 84.2% on the object category given the input. |
Adversarial Evaluation of Multimodal Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Existing evidence that visual context helps multimodal machine translation systems is unconvincing due to inconsistencies between text-similarity metrics and human judgements. |
| Approach: | They propose an adversarial evaluation method to examine the utility of image data in multimodal machine translation. |
| Outcome: | The proposed method shows that only one out of three publicly available systems is sensitive to this perturbation of the data. |
Polish Corpus of Annotated Descriptions of Images (L18-1)
Copied to clipboard
| Challenge: | a new dataset of image descriptions is presented in Polish . the dataset is too small for training a sophisticated language-vision system. |
| Approach: | They propose to use a Polish dataset to analyze image descriptions . the descriptions are morphosyntactically analysed and annotated by human annotators . |
| Outcome: | The proposed model learns about the inter-modal correspondences between language and vision. |
R2-MultiOmnia: Leading Multilingual Multimodal Reasoning via Self-Training (2025.acl-long)
Copied to clipboard
| Challenge: | Recent studies have introduced eclectic strategies to enhance MLLMs’ reasoning capabilities, but they remain related to a single language. |
| Approach: | They propose a modular approach that instructs models to abstract key elements of the reasoning process and refine reasoning trajectories via self-correction. |
| Outcome: | The proposed approach improves multimodal reasoning, gets aligned performances among the languages approaching strong models and improves the model's performance. |
Learning to Describe Differences Between Pairs of Similar Images (D18-1)
Copied to clipboard
| Challenge: | Using crowd-sourced annotations, we generate text descriptions of differences between two images . we use a dataset to generate concise and fluent descriptions of visual data . |
| Approach: | They propose a task of automatically generating text to describe the differences between two images . they crowd-sourced the difference descriptions for pairs of images extracted from video-surveillance footage . |
| Outcome: | The proposed model outperforms models that use attention alone for single-sentence generation and multi-sentent generation. |
Obtaining Faithful Interpretations from Compositional Neural Networks (2020.acl-main)
Copied to clipboard
Sanjay Subramanian, Ben Bogin, Nitish Gupta, Tomer Wolfson, Sameer Singh, Jonathan Berant, Matt Gardner
| Challenge: | Neural module networks (NMNs) are a popular approach for modeling compositionality but prior work implicitly assumed that the structure of the network modules provides a faithful explanation of the model’s reasoning. |
| Approach: | They propose to use auxiliary supervision to train a model with a structured model that can understand the reasoning process and make better choices for module architecture. |
| Outcome: | The proposed models on two datasets show that the proposed models do not provide a faithful explanation of model behaviour. |
Segmented Recurrent Transformer: An Efficient Sequence-to-Sequence Model (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Transformers have shown dominant performance across a range of domains including language and vision, but their computational cost grows quadratically with the sequence length, making their usage prohibitive for resource-constrained applications. |
| Approach: | They propose a segmented recurrent transformer that combines segmente recursion with recursive attention to reduce the computational cost. |
| Outcome: | The proposed model achieves higher ROUGE1 scores and lower computational complexity than current approaches. |
InstructPart: Task-Oriented Part Segmentation with Instruction Reasoning (2025.acl-long)
Copied to clipboard
Zifu Wan, Yaqi Xie, Ce Zhang, Zhiqiu Lin, Zihan Wang, Simon Stepputtis, Deva Ramanan, Katia P. Sycara
| Challenge: | Large multimodal foundation models perceive objects as indivisible, overlooking the components that constitute them. |
| Approach: | They propose a novel benchmark for large multimodal foundation models comprising hand-labeled part segmentation annotations and task-oriented instructions to evaluate their performance. |
| Outcome: | The proposed benchmark improves performance of current models in understanding and executing part-level tasks within everyday contexts. |